Automatic Capacity Tuning of Very Large Vc-dimension Classiers

Large VC-dimension classiers can learn dicult tasks, but are usually impractical because they generalize well only if they are trained with huge quantities of data. In this paper we show that even very high-order polynomial classiers can be trained with a small amount of training data and yet generalize better than classiers with a smaller VC-dimension. This is achieved with a maximum margin algorithm (the Generalized Portrait). The technique is applicable to a wide variety of classiers, including Per-ceptrons, polynomial classiers (sigma-pi unit networks) and Radial Basis Functions. The eective number of parameters is adjusted automatically by the training algorithm to match the complexity of the problem. It is shown to equal the number of those training patterns which are closest patterns to the decision boundary (supporting patterns). Bounds on the generalization error and the speed of convergence of the algorithm are given. Experimental results on handwritten digit recognition demonstrate good generalization compared to other algorithms.